Data Validation: Heliconia Demography Project

We use the R package pointblank to review and validate the demographic data set (HDP_1997_2009.csv) and plot-level descriptors (HDP_plots.csv) published in Bruna et al. (2023) and archived in the Dryad online data repository. The report below includes:

1. the different validation tests that were conducted,
2. the date of the most recent test,
3. each test’s criteria for ‘pass’, ‘warn’ and ‘stop’,
4. the number of ‘units’ (i.e., rows or columns) assessed in each test,
5. how many of these units passed or failed, and
6. a button for downloading a .csv file of the records flagged by a particular validation test. Note that these are not necessarily errors. For instance, the validation procedure for ‘plant size - height’ returns as ‘stop’ all plants >2 m tall. Heliconia plants can exceed this threshold; this test is simply designed to flag any such individuals. In contrast, the data set should not have any duplicated rows. A notification of ‘fail’ for this test indicates an error that can be corrected by downloading the csv file, reviewing the duplicated rows, and uploading the necessary corrections.

---

Dataset Structure: Data types

Tests to determine if columns are correctly coded as integer, character, etc.
Test criteria: Strict (‘stop’ if any rows fail).

Pointblank Validation
Data Validation tibbleWARN 1 STOP 0.02 NOTIFY —
		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	S	N	EXT
	1	Height is measured to nearest cm col_vals_expr()	—	ht%%1 == 0		✓	57K	57K 1.00	0 0.00	—	○	—	—
	2	Shoots is interger col_vals_expr()	—	shts%%1 == 0		✓	57K	57K 1.00	0 0.00	—	○	—	—
	3	Number of inflorescences is integer col_vals_expr()	—	infl%%1 == 0		✓	2K	2K 1.00	0 0.00	—	○	—	—
2023-05-26 15:38:08 UTC < 1 s 2023-05-26 15:38:08 UTC

---

Dataset Structure: Plot & Subplot IDs

Test for any nonexistent values of plot_id (e.g., ‘FF-10’, ‘CF-23’) or subplot (e.g., ‘H23’, ‘A11’).
Test criteria: Strict (‘stop’ if any rows fail).

Pointblank Validation
Data Validation tibbleWARN 1 STOP 0.02 NOTIFY —
		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	S	N	EXT
	1	col_vals_in_set()	▮plot_id	CF-1, CF-2, CF-3, CF-4, CF-5, CF-6, FF-1, FF-2, FF-3, FF-4, FF-5, FF-6, FF-7		✓	67K	67K 1.00	0 0.00	—	○	—	—
	2	col_vals_in_set()	▮subplot	A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, E1, E2, E3, E4, E5, E6, E7, E8, E9, E10, F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, G1, G2, G3, G4, G5, G6, G7, G8, G9, G10, H1, H2, H3, H4, H5, H6, H7, H8, H9, H10, I1, I2, I3, I4, I5, I6, I7, I8, I9, I10, J1, J2, J3, J4, J5, J6, J7, J8, J9, J10		✓	67K	67K 1.00	0 0.00	—	○	—	—
2023-05-26 15:38:09 UTC < 1 s 2023-05-26 15:38:09 UTC

Dataset Structure: Duplicated or Missing Values

Tests for duplicated rows, missing plant_ID numbers, or duplicate plant_id numbers (test is done for every survey year).
Test criteria: Strict (‘stop’ if any rows fail).

Pointblank Validation
Data Validation tibbleWARN 1 STOP 0.02 NOTIFY —
		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	S	N	EXT
	1	duplicated rows rows_distinct()	—	—		✓	67K	67K 1.00	0 0.00	—	○	—	—
	2	col_vals_not_null()	▮plant_id	—		✓	67K	67K 1.00	0 0.00	—	○	—	—
	3	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	3K	3K 1.00	0 0.00	—	○	—	—
	4	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	4K	4K 1.00	0 0.00	—	○	—	—
	5	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	5K	5K 1.00	0 0.00	—	○	—	—
	6	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	6K	6K 1.00	0 0.00	—	○	—	—
	7	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	6K	6K 1.00	0 0.00	—	○	—	—
	8	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	6K	6K 1.00	0 0.00	—	○	—	—
	9	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	6K	6K 1.00	0 0.00	—	○	—	—
	10	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	6K	6K 1.00	0 0.00	—	○	—	—
	11	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	7K	7K 1.00	0 0.00	—	○	—	—
	12	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	5K	5K 1.00	0 0.00	—	○	—	—
	13	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	6K	6K 1.00	0 0.00	—	○	—	—
	14	Check for duplicate ID's within each year rows_distinct()	▮plant_id	—		✓	6K	6K 1.00	0 0.00	—	○	—	—
2023-05-26 15:38:10 UTC 2.4 s 2023-05-26 15:38:12 UTC

Plant Characteristics: Size & Flowering

Tests to determine how many values of plant size (shts, ht) or infloresence number (infl) are outside the range of most values.
Test criteria: ‘warn’ if \(\geq\) 1 rows fail conditions, ‘stop’ if \(\geq\) 2% of rows fail conditions.

Pointblank Validation
Data Validation tibbleWARN 1 STOP 0.02 NOTIFY —
		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	S	N	EXT
	1	shoots between 0 and 20 col_vals_between()	▮shts	[0, 20]		✓	67K	67K 0.99	8 0.01	●	○	—	CSV
	2	height between 0 and 200cm col_vals_between()	▮ht	[0, 200]		✓	67K	67K 0.99	2 0.01	●	○	—	CSV
	3	infloresences between 0 and 3 col_vals_between()	▮infl	[0, 3]		✓	67K	67K 0.99	15 0.01	●	○	—	CSV
2023-05-26 15:38:13 UTC < 1 s 2023-05-26 15:38:13 UTC

Plant Characteristics: Growth

Tests for unusual changes in plant size from \[Year_{t}\] to \[Year_{t+1}\]. Conducted for both height and shoot number. Test criteria: ‘warn’ if \(\geq\) 1 rows fail conditions, ‘stop’ if \(\geq\) 2% of rows fail conditions.

Pointblank Validation
Check growth & regression tibbleWARN 1 STOP 0.02 NOTIFY —
		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	S	N	EXT
	1	|% change in height| < 200% col_vals_lt()	▮ht_pc	2		✓	67K	66K 0.99	420 0.01	●	○	—	CSV
	2	|∆ height| < 100cm col_vals_between()	▮ht_diff	[−100, 100]		✓	67K	67K 0.99	11 0.01	—	●	—	CSV
	3	|∆ shoot number| < 5 col_vals_between()	▮shts_diff	[−5, 5]		✓	67K	67K 0.99	201 0.01	—	●	—	CSV
2023-05-26 15:38:14 UTC < 1 s 2023-05-26 15:38:14 UTC

Seedlings: Initial size

Tests for seedlings whose size at initial marking was unusually large. Conducted for both height and shoot number.
Test criteria: ‘warn’ if \(\geq\) 1 rows fail conditions, ‘stop’ if \(\geq\) 2% of rows fail conditions.

Pointblank Validation
Check seedlings tibbleWARN 1 STOP 0.02 NOTIFY —
		STEP	COLUMNS	VALUES	TBL	EVAL	UNITS	PASS	FAIL	W	S	N	EXT
	1	shoots < 3 col_vals_lt()	▮shts	3		✓	3K	3K 0.99	12 0.01	●	○	—	CSV
	2	height < 30cm col_vals_lt()	▮ht	30		✓	3K	3K 0.99	3 0.01	●	○	—	CSV
2023-05-26 15:38:15 UTC < 1 s 2023-05-26 15:38:15 UTC

---

Missing values: Height